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(54) Information processing apparatus and methods, and information providing media 



(57) An information processing apparatus (1) com- 
prises an image-sensing means (23) for sensing an im- 
age of a subject, an extracting means (63) for extracting 
predetermined feature data from the image sensed by 



the image-sensing means (23), a setting means (57) for 
setting sound data to be reproduced and a reproducing 
means (8, 60) for reproducing the sound data set by the 
setting means (57) according to the feature data extract- 
ed by the extracting means (63). 
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Description 

[0001] The present invention relates to an information 
processing apparatus, an information processing meth- 
od, and an information providing medium. s 
[0002] Conventional sound reproducing systems in- 
clude a record player, a reproducing device using an op- 
tical disc, and a cassette tape recorder. These sound 
reproducing systems reproduce sound data recorded in 
advance on a recording medium. io 
[0003] Recently, users not satisfied with the simple re- 
production of recorded sound data are increasingly turn- 
ing to so-called computer music in which, for example, 
music is played by use of hardware and software and 
played music is recorded on a recording medium. Com- is 
puter music also involves the automatic play of musical 
instruments. In the automatic play, recorded MIDI (Mu- 
sical Instruments Digital Interface) sequence data for 
sound reproduction is supplied to a sound generator for 
sound output. 20 
[0004] The above-mentioned computer music is 
based on a personal computer. Music is played and au- 
tomatic performance is executed by operating the 
mouse, keyboard, touch panel, and other man-machine 
interfaces provided by the personal computer. Conse- 25 
quently, the performance of computer music requires in- 
put devices that the user can operate directly with the 
hand. This makes the above-mentioned computer mu- 
sic systems unsuitable for the enjoyment of live perform- 
ance for example in which performers and audiences 30 
enjoy the playing of music involving movement. 
[0005] Generally, playing music and execution of au- 
tomatic performance require special knowledge and 
techniques. Therefore, the practice of computer music 
requires specialists. Amateurs can only listen to repro- 35 
duced music. However, some amateurs desire to ar- 
range music on their own in a simple way. 
[0006] According to a first aspect of the present inven- 
tion, there is provided an information processing appa- 
ratus including an image-sensing means for sensing an *o 
image of a subject, an extracting means for extracting 
predetermined feature data from the image sensed by 
the image-sensing means, a setting means for setting 
sound data to be reproduced and a reproducing means 
for reproducing the sound data set by the setting means 45 
according to the feature data extracted by the extracting 
means. 

[0007] According to a second aspect of the present 
invention, there is provided an information processing 
method including the steps of image-sensing an image s° 
of a subject, extracting predetermined feature data from 
the image sensed in the image-sensing step, setting 
sound data to be reproduced and reproducing the sound 
data set in the setting step according to the feature data 
extracted in the extracting step. 55 
[0008] According to third and fourth aspects of the 
present invention, there are provided an information 
providing medium for providing a program, and/or a pro- 
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gram itself, readable by a computer for making an infor- 
mation processing apparatus execute processing in- 
cluding the steps of image-sensing an image of a sub- 
ject, extracting predetermined feature data from the im- 
age sensed in the image-sensing step, setting sound 
data to be reproduced and reproducing the sound data 
set in the setting step according to the feature data ex- 
tracted in the extracting step. 

[0009] According to a preferred embodiment of the in- 
vention, an image of a subject is sensed, predetermined 
feature data is extracted from the sensed image, and 
sound data is reproduced according to the extracted 
feature data. This allows the user to arrange music only 
by executing simple setting operations. 
[0010] The preferred embodiment of the present in- 
vention generates a sound and achieves changes to the 
motion and shape of an object displayed on the screen 
by changing the image sensed by a CCD (Charge Cou- 
pled Device) video camera for example. 
[0011] The invention will now be described by way of 
example with reference to the accompanying drawings, 
throughout which like parts are referred to by like refer- 
ences, and in which: 

FIG. 1 is a perspective view illustrating an exempla- 
ry portable personal computer to which the present 
invention may be applied, with the display unit 
raised; 

FIG. 2 is a top view of the portable personal com- 
puter of FIG. 1; 

FIG. 3 is a left side view of the portable personal 
computer of FIG. 1 with the display unit closed; 
FIG. 4 is a right side view of the portable personal 
computer of FIG. 1 with the display unit raised to 
180 degrees from the top of the main frame; 
FIG. 5 is a front view of the portable personal com- 
puter of FIG. 3; 

FIG. 6 is a bottom view of the portable personal 
computer of FIG. 4; 

FIG. 7 is a block diagram illustrating an exemplary 
constitution of the circuitry of the portable personal 
computer of FIG. 1; 

FIG. 8 is a diagram illustrating an example of screen 
displayed on the display unit; 
FIG. 9 is a flowchart for describing processing to be 
performed by the user; 

FIG. 10 is a diagram illustrating another example of 
screen displayed on the display unit; 
FIGS. 11 A, 11B, 11C and 11 D are diagrams illus- 
trating examples of display in the setting box shown 
in the screen of FIG. 8; 

FIG. 12 is a diagram illustrating still another exam- 
ple of screen displayed on the display unit; 
FIG. 13 is a diagram illustrating a pointer and bub- 
bles; 

FIGS. 14A, 14B and 14C are diagrams illustrating 
examples of display in the setting box in the screen 
of FIG. 8; 
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FIG. 15 is a diagram illustrating yet another exam- 
ple of screen displayed on the display unit; 
FIGS. 16A, 16B and 16C are diagrams illustrating 
motions of sound objects; 

FIG. 17 is a flowchart for describing processing to 
be executed for displaying sound objects; and 
FIG. 18 is a flowchart for describing processing to 
be executed for a sound object to generate a sound. 

[001 2] This invention will be described in further detail 
by way of example with reference to the accompanying 
drawings. In order to clarify the correspondence be- 
tween the claimed means of the invention and the fol- 
lowing embodiment, each of these means is followed by 
an example of corresponding embodiment in parenthe- 
ses. Obviously, this description will not in any manner 
restrict each means to the corresponding embodiment 
mentioned in parentheses. 

[0013] An information processing apparatus accord- 
ing to claim 1 hereto comprises an image-sensing 
means (for example, a CCD video camera 23 shown in 
FIG. 1 ) for sensing an image of a subject, an extracting 
means (for example, step S22 shown in FIG. 18) for ex- 
tracting predetermined feature data from the image 
sensed by the image-sensing means, a setting means 
(for example, step S2 shown in FIG. 9) for setting sound 
data to be reproduced, and a reproducing means (for 
example, step S25 shown in FIG. 18) for reproducing 
the sound data set by the setting means according to 
the data extracted by the extracting means. 
[0014] An information processing apparatus accord- 
ing to claim 2 comprises a parameter setting means (for 
example, step S3 shown in FIG. 9) for setting parame- 
ters for controlling the motion of an object generated in 
response to the sound data set by the setting means 
and a display control means (for example, steps S12 
and S1 3 shown in FIG. 1 7) for controlling the display of 
the object. 

[0015] An information processing apparatus accord- 
ing to claim 3 comprises a recording means (for exam- 
ple, a HDD 56 shown in FIG. 6) for recording the data 
set by the setting means and the parameter setting 
means. 

[0016] FIGS. 1 through 6 illustrate an exemplary con- 
stitution of a portable personal computer practiced as 
one preferred embodiment of the invention. In the fig- 
ures, the personal computer 1 is of mini-note type, which 
is basically composed of a main frame 2 and a display 
block 3 pivotally mounted thereon. FIG. 1 perspectively 
illustrates the personal computer 1 with the display block 
3 open relative to the main frame 2. FIG. 2 is a top view 
of the personal computer 1 shown in FIG. 1. FIG. 3 is a 
left side view illustrating the personal computer 1 shown 
in FIG. 1 with the display block 2 closed against the main 
frame 2. FIG. 4 is a right side view illustrating the per- 
sonal computer 1 shown in FIG. 1 with the display block 
3 open by 1 80 degrees relative to the main frame 2. FIG. 
5 is a top view illustrating the personal computer 1 



shown in FIG. 3. FIG. 6 is a bottom view illustrating the 
personal computer 1 shown in FIG. 4. 
[001 7] The main frame 2 is arranged on the top there- 
of with a keyboard 4 that is operated to enter various 

5 characters and symbols and a Track Point (trademark) 
5 that is operated to move the mouse cursor for exam- 
ple. The main frame 2 is further arranged on the top 
thereof with a speaker 8 for outputting sound and a shut- 
ter button 1 0 that is operated for image-sensing th rough 

to a CCD video camera 23 disposed on the display block 
3. 

[0018] A claw 1 3 is disposed on the upper end of the 
display block 3. A hole 6 in which the claw 1 3 mates is 
disposed on the main frame 2 at a position that corre- 

75 sponds to the position of the claw 13 when the display 
block 3 is closed against the main frame 2. A slide lever 
7 is disposed on the front face of the main frame 2 in a 
movable manner along the front face. The slide lever 7 
is adapted to latch and unlatch the claw 1 3 mated in the 

20 hole 6. In the unlocked state, the display block 3 can be 
pivotally moved relative to the main frame 2. A micro- 
phone 24 is disposed beside the claw 1 3. As shown in 
FIG. 6, the microphone 24 can also pick up sound com- 
ing from the back of the personal computer 1 . 

25 [0019] The front face of the main frame 2 is also dis- 
posed with a programmable power key (PPK) 9. On the 
right-side face of the main frame 2, an exhaust port 11 
is disposed as shown in FIG. 4. On the lower portion of 
the front face of the main frame 2, an intake port 14 is 

30 disposed as shown in FIG. 5. To the right of the exhaust 
port 1 1 , a slot 1 2 is disposed for accommodating a PC- 
MCIA (Personal Computer Memory Card International 
Association) card (a PC card in short). 
[0020] On the top face of the display block 3, an LCD 

35 (Liquid Crystal Display) 21 is disposed for displaying im- 
ages. On the upper end of the display block 3, an image- 
sensing block 22 is disposed in a pivotally movable man- 
ner relative to the display block 3. To be more specific, 
the image-sensing block 22 can pivotally move to any 

40 position in a range of 1 80 degrees at right angles to the 
vertical direction of the display block 3. The image-sens- 
ing block 22 has the CCD video camera 23. 
[0021] In the lower portion of the display block 3, a 
power light PL, a battery light BL, a message light ML, 

45 and other light or lights each constituted by a LED (Light 
Emitting Diode) are arranged, facing the main frame 2. 
Reference numeral 40 shown in FIG. 3 denotes a power 
switch disposed on the left side face of the main frame 
2. Reference numeral 25 shown in FIG. 5 denotes an 

so adjustment ring for adjusting focus of the CCD video 
camera 23. Reference numeral 26 shown in FIG. 6 de- 
notes a cover for an opening through which an add-on 
memory is installed in the main frame 2. Reference nu- 
meral 41 denotes a hole through which a pin is inserted 

55 to unlatch a claw locking the cover 26 to the main frame 
2. 

[0022] FIG. 7 exemplifies the internal constitution of 
the personal computer 1. As shown, an internal bus 51 
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is connected to a CPU (Central Processing Unit) 52, a 
PC card 53 that is inserted as required, a RAM (Random 
Access Memory) 54, and a graphics chip 81 . The inter- 
nal bus 51 is also connected to an external bus 55. The 
external bus 55 is connected to the hard disk drive s 
(HDD) 56, an I/O (Input/Output) controller 57, a key- 
board controller 58, a Track Point controller 59, a sound 
chip 60, an LCD controller 83, and a modem 50. 
[0023] The CPU 52 controls the above-mentioned 
components of the personal computer 1 . The PC card to 
53 is inserted to add an optional capability. 
[0024] The RAM 54 stores, when the personal com- 
puter 1 starts, an electronic mail program (an application 
program) 54A, an auto pilot program (an application pro- 
gram) 54B, and an OS (Operating System) 54C from 15 
the HDD 56. 

[0025] The electronic mail program 54A handles elec- 
tronic messages transferred from a network through a 
communication line like telephone line. The electronic 
mail program 54 A has an in -coming mail capturing ca- 20 
pability as a particular capability. The in-coming mail 
capturing capability checks a mail box 93 A of a mail 
server 93 for a mail addressed to that user and, if such 
a mail is found, captures the same. 

[0026] The auto pilot program 54B sequentially starts 25 
plural preset processing operations (or programs) in a 
predetermined order. 

[0027] The OS 54C controls basic computer opera- 
tions exemplified by Windows 95 (trademark). 
[0028] The HDD 56 on the external bus 55 stores an 30 
electronic mail program 56 A, an auto pilot program 56 B, 
and an OS 56C. These programs are sequentially sent 
into the RAM 54 at the time of booting-up. 
[0029] The I/O controller 57 has a microcontroller 61 
provided with an I/O interface 62. The microcontroller 35 
61 is constituted by the I/O interface 62, a CPU 63, a 
RAM 64, and a ROM (Read Only Memory) 69 intercon- 
nected with each other. The RAM 64 has a key-input 
status register 65, a LED control register 66, a setting 
time register 67, and a register 68. The setting time reg- *o 
ister 67 is used to start a boot sequence controller 76 
when a time (or a boot condition) set by user comes. 
The register 68 holds the correspondence between a 
preset operator key combination and an application pro- 
gram to be started. When the user enters this operator 45 
key combination, the corresponding application pro- 
gram (for example, the electronic mail program) starts. 
[0030] The key-input status register 65 holds an op- 
erator key flag when the PPK 9 for single-touch opera- 
tion is pressed. The LED control register 66 controls the so 
turn-on/oft of the message light ML that indicates the 
operating state of the application program (the electron- 
ic mail program) held in the register 68. The user can 
set any desired time to the time setting register 67. 
[0031] A backup battery 74 is connected to the micro- 55 
controller 61 , thereby preventing the values set to the 
registers 65, 66, and 67 from being cleared after the 
main frame 2 is powered off. 



[0032] The ROM 69 in the microcontroller 61 stores a 
wakeup program 70, a key-input monitor program 71, 
and an LED control program 72 in advance. The ROM 
69 is constructed of an EEPROM (Electrically Erasable 
and Programmable Read Only Memory) for example. 
The EEPROM is known as a flash memory. An RTC (Re- 
al Time Clock) 75A for always counting current time is 
also connected to the microcontroller 61 . 
[0033] The wakeup program 70 stored in the ROM 69 
checks, based on the current time data supplied from 
the RTC 75, whether the time preset to the setting time 
register 67 has been reached. If the time is found 
reached, the wakeup program 70 starts a predeter- 
mined processing operation (or a predetermined pro- 
gram). The key-input monitor program 71 monitors the 
pressing of the PPK 9 by the user. The LED control pro- 
gram 72 controls the turn-on/off of the message light ML. 
[0034] The ROM 69 also stores a BIOS (Basic Input/ 
Output System) 73. The BIOS is a software program for 
controlling the transfer of data between the OS or an 
application software program and peripheral devices 
(the display monitor, the keyboard, and the hard disk 
drive). 

[0035] The keyboard controller 58 connected to the 
external bus 55 controls the input made on the keyboard 
4. The Track Point controller 59 controls the input made 
on the Track Point 5. 

[0036] The sound chip 60 captures the input from the 
microphone 24 and supplies an audio signal to the built- 
in speaker 8. 

[0037] The modem 50 connects the personal compu- 
ter 1 to a communication network 92 such as the Internet 
or the mail server 93 through a public telephone line 90 
or an Internet service provider 91 . 
[0038] Image data captured by the CCD video camera 
23 is processed in a processing block 82 to be supplied 
to the graphics chip 81 connected to the internal bus 51 . 
The graphics chip 81 stores the video data inputted from 
the CCD video camera 23 through the processing block 
82 into a built-in VRAM (Video RAM) 81 A and reads the 
stored video data as required and outputs the same to 
the LCD controller 83. The LCD controller 83 outputs 
the video data supplied from the graphics chip 81 for 
display. A back light 84 illuminates the LCD 21 from be- 
hind the same. 

[0039] The power switch 40 turns on/off the power to 
the personal computer 1. A half-press switch 85 is 
turned on when the shutter button 10 is pressed to the 
half position. A full-press switch 86 is turned on when 
the shutter button 10 is fully pressed. A reverse switch 
87 is turned on when the image-sensing block 22 is ro- 
tated 180 degrees (namely, when the CCD video cam- 
era 23 is rotated in the direction behind the LCD 21). 
[0040] FIG. 8 illustrates one example of a screen to 
be displayed on the LCD 21 . Shown in this screen are 
a music composing window 1 1 0 and a sound file window 
120. The music composing window 110 opens when 
music is composed by use of a sound file selected in the 
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sound file window 1 20 and an image sensed by the CCD 
camera 23. 

[0041] The music composing window 110 is made up 
of a selecting block 111 for changing the size or dis- 
played contents of this window, an image block 112 for 
displaying an image sensed by the CCD video camera 
23, a setting block 113 for setting the display of the im- 
age block 112 and the motion of a sound object (to be 
described later) to be displayed on a stage 115, and a 
command button 114 which is operated mainly when 
switching between the images of the setting block 113. 
[0042] "File" in the selecting block 111 is operated to 
record the settings in this window to the HDD 56 or read 
data from the same. "Display" is operated to change the 
display screen setup of the music composing window 
110 for example. "Help" is operated to get information 
about the operations of this system. When "File", "Dis- 
play" and "Help" are operated, pull-down menus open. 
The three small boxes in the upper right corner of the 
selecting block 1 1 1 are used to expand or shrink the size 
of the music composing window 1 10 or close the same. 
[0043] The image block 112 displays an image 
sensed by the CCD camera 23 or a grid mesh according 
to the data set in the setting block 113. In the display 
example of FIG. 8, the image shown is a person holding 
a light emitting object like a flashlight. 
[0044] The setting block 113 sets the display of the 
image block 112 and shows screen for setting the mo- 
tion of a sound object displayed on the stage 1 1 5 to be 
described later. Display examples of the setting block 
113 will be described with reference to FIGS. 11 A 
through 11 D and FIGS. 14A through 14C. 
[0045] Command button 114 "PLAY" is operated 
when the settings have all been made, creating a sound 
(tone). Command button 114 "EDIT" is operated to dis- 
play a screen in the setting block 11 3 for setting condi- 
tions (or parameters) for sounding the created sound. 
Command button 1 1 4 "Object" is operated to set param- 
eters associated with the motion of a sound object to be 
displayed on the stage 115. 

[0046] The stage 115 displays a sound object corre- 
sponding to a sound file selected in the sound file win- 
dow 1 20 by the user. The displayed sound object moves 
on the stage 115 according to the data set in the setting 
block 113. 

[0047] The sound file window 1 20 is made up of a se- 
lecting block 121 and a file display block 122. The se- 
lecting block 121 is generally the same in constitution 
and operation as the selecting block 111. Therefore, the 
description of the selecting block 121 is skipped. The 
file display block 122 displays three sound file icons 
123-1 through 123-3 (hereafter, these icons are gener- 
ically refereed to simply as icon 123 if the distinction is 
not required). The files represented by these icons are 
named "SOUND 1", "SOUND 2" and "SOUND 3" re- 
spectively. 

[0048] Each sound file contains PCM (Pulse Code 
Modulation) sound data such as of AIFF (Audio Inter- 



change File Format) and WAVE (Waveform audio) for- 
mat and data captured by MIDI for example. In addition, 
data recorded on a compact disc can be used as a 
sound file. 

5 [0049] A cursor 1 30 moves in response to the opera- 
tion of the Track Point 5 operated by the user. 
[0050] It should be noted that the screen shown in 
FIG. 8 is exemplary and therefore another option may 
be provided to the selecting block 111 (or the selecting 
io block 121) the options may be represented by icons. 
[0051] The following describes, with reference to the 
flowchart of FIG. 9, the settings to be made by the user. 
In step S1, the user selects one sound file from the 
sound files (represented by icon 1 23) displayed in the 

is file display block 1 22 of the sound file window 1 20. This 
selection is made by moving the cursor 130 to the icon 
123 of a desired sound file, dragging the selected icon 
123, and dropping the same onto the stage 115 of the 
music composing window 110. 

20 [0052] FIG. 10 exemplifies a case in which the icon 
123 has been selected as described above. The icon 
123 dropped on the stage 115 is then displayed as a 
sound object 141 different in shape from the icon 123. 
In this example, the sound object 141 is shown in the 

25 shape of a musical note. 

[0053] The sound object 1 41 may be a default picture 
imparted when the icon 1 23 has been dropped onto the 
stage 115, a picture created by the user, or an image 
captured from a digital camera for example. In this ex - 

30 ample, the stage 115 has no background picture. The 
user can set a desired picture as the background. The 
user can perform these settings by operating "Display" 
of the selecting block 111 and selecting and setting a 
necessary item of the pulldown menu. Alternatively, the 

35 user can select and set a necessary item by clicking the 
stage 1 1 5 by the right-side button of mouse. When the 
stage 115 is thus clicked, a pulldown menu appears in 
which the user selects a background picture in a dialog 
box displayed. 

40 [0054] When the sound file selection is completed in 
step S1 , then edit setting is made in step S2. The edit 
setting is effected by operating the command button 114 
"EDIT" by use of the cursor 1 30. When the "EDIT" button 
is operated, a screen as shown in FIG. 11 A appears in 

45 the setting block 113. 

[0055] FIG. 11 A illustrates a setting screen for chang- 
ing the motion and sound of the sound object 141 by . 
brightness. A matrix 1 50 composed of 9 squares shown 
in the upper left of the screen and. the numbers 0 

50 through 8 attached to these squares denote that the im- 
age block 112 is equally divided by 9. To be more spe- 
cific, when the image shown in FIG. 11 A is displayed in 
the setting block 113 as shown in FIG. 12, a grid is 
shown in the image block 112 to indicate that the image 

55 block 112 is divided into 9 equal portions. Each square 
making up the grid is hereafter referred to as a mesh as 
appropriate. 

[0056] A brightness setting block 1 51 is made up of 9 
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bars numbered in correspondence to the matrix 1 50 and 
one brightness reference bar. The brightness reference 
bar is shown in gradation at the left end of the brightness 
setting block 151. The user references this bar to select 
a desired brightness. 

[0057] In the screen shown in FIG. 1 1 A, the user sets 
a brightness threshold. To be more specific, the user ref- 
erences the brightness reference bar, determines a box 
at desired brightness of the bar having the number cor- 
responding to the mesh to be sounded, and clicks the 
selected box. 

[0058] The example of FIG. 11 A illustrates a state in 
which the brightest portion of the bar corresponding to 
the square 0 in the matrix 150 has been clicked for se- 
lection. The selected box is colored. In other words, any 
bar having no setting of brightness threshold has no 
colored box. It should be noted that, for one sound object 
141 , brightness thresholds may be set to plural bars. 
[0059] Below the matrix 1 50, a page display block 1 52 
is located for showing a page number. This brightness 
setting screen is page 1 for example. To the left of the 
page display block 152, a previous page display button 
1 53 is located. To the right of the page display block 1 52, 
a next page display button 154 is located. 
[0060] When the brightness has been set as de- 
scribed above, the user operates the next page display 
button 154, upon which a setting screen as shown in 
FIG. 11 B is displayed in the setting block 113. In the 
newly displayed setting screen, the user sets a virtual 
space of the stage 1 1 5. "PERSPECTIVE" sets the stage 
115 into a virtual three-dimensional space. Namely, the 
sound object 1 41 displayed on the stage 1 1 5 moves hor- 
izontally, vertically, and in depth direction in the virtual 
three-dimensional space. "PLANE" sets the stage 115 
into a two-dimensional space. Namely, the sound object 
141 displayed on the stage 115 moves horizontally and 
vertically in the two-dimensional space. 
[0061] When moving on a same plane (or a two-di- 
mensional space) horizontally or vertically, the sound 
object 141 does not change its size. However, when 
moving in the three-dimensional space, the sound ob- 
ject 141 increases its size as it comes forward and re- 
duces its size as it goes into depth. The example of FIG. 
1 1 B shows a state in which "PERSPECTIVE" as a three- 
dimensional space is selected. 

[0062] When the user operates the next page display 
button 154, a setting screen as shown in FIG. 11 C is 
displayed. In the newly displayed setting screen, the us- 
er sets a direction in which the sound object 141 starts 
moving (that is, an initial value) when command button 
1 1 4 "PLAY" is operated. In this example, the initial value 
is set so that the sound object 141 moves upward. 
[0063] When the user operates the next page display 
button 154, a setting screen as shown in FIG. 11 D is 
displayed. In this newly displayed screen, the user sets 
whether a bubble is to be generated or not. If a bubble 
is to be generated, then the user sets whether the bub- 
ble is to be generated continuously or randomly. In the 



example of FIG. 11D, generation of a bubble is set and 
the generation is made randomly. 
[0064] When the bubble generation is set, a pointer 
160 is displayed on the stage 115 as shown in FIG. 13. 

5 The pointer 160 is displayed such that it moves in re- 
sponse to a portion of the image in the image block 1 1 2 
for which the motion vector is found fastest; for example, 
in response to the motion of a hand if the image shown 
in the image block 112 is a person waving his or her 

to hand. The pointer 160 is so called because it points at 
a fastest-moving object. 

[0065] The pointer 1 60 may take any shape and color. 
In the example of FIG. 13, the pointer 160 is spherical. 
From this pointer 160, spherical objects called bubbles 

is are generated continuously or randomly. Bubbles are al- 
so generated from screen frames (walls) of the stage 
115. When the sound object 141 hits one of these bub- 
bles, the sound object 141 bounces from the bubble. 
The bubbles are adapted to hit the sound object 141, 

20 get out of the stage 115 through its walls, or disappear 
when a predetermined time has passed. 
[0066] Now, returning to the flowchart of FIG. 9, when 
the user has completed the above-mentioned various 
setting operations in step S2, then, in step S3, the user 

25 sets a motion of the sound object 141. This setting starts 
by operating the command button 114 "Object". When 
the "Object" button is pressed, a screen shown in FIG. 
14A is displayed in the setting block 113. 
[0067] In the setting screen shown in FIG. 14A, the 

30 user sets a parameter for determining the motion of the 
sound object 141. By "FRICTION", the user sets the fric- 
tion between the sound object 141 and the stage 115. 
As the friction increases, the sound object 141 stops 
soon after it starts moving. As the friction decreases, the 

35 sound object 141 will not stop soon once it starts mov- 
ing. 

[0068] By "MASS (BOUNCE)", the user sets whether 
the sound object 141 is to have a mass or not. By clicking 
radio button "ON", the user can give a mass to the sound 
40 object 1 41 . The sound object 141 given a mass bounces 
from another sound object or a bubble when hit by it 
("bounce" means a change in direction in which the 
sound object 141 travels). 

[0069] On the other hand, if the user sets that the 
45 sound object 141 is to be given no mass (that is, if the 
user clicks radio button "OFF"), hitting of another sound 
object or a bubble against the sound object 141 does 
not make the same bounce or the amount of bounce is 
small. 

50 [0070] When the user has completed these setting 
operations and presses the next page display button 
154, a screen as shown in FIG. 14B is displayed in the 
setting block 113. In this screen, the user sets a time in 
which a tone is sounded. Namely, since the sound object 

55 141 js set so that a tone is sounded when a predeter- 
mined mesh of the image block 112 has reached a pre- 
determined brightness, a sound length is set in this 
screen. 
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[0071] In the example of FIG. 14B, the sound length 
is adapted to be set to 1 to 5 seconds. When the corre- 
sponding mesh has reach a predetermined brightness, 
the sound object 141 sounds by the number of seconds 
set in this screen. The example of FIG. 14B shows a 5 
state in which the button is clicked on 5-second position 
and sounding is on. 

[0072] When the user has completed the sound 
length setting operation and operates the next page dis- 
play button 154, a screen as shown in FIG. 14C is dis- 10 
played. In this screen, the user sets a motion of the 
sound object 1 41 against the pointer 1 60. When the user 
turns on radio button "Follow", the sound object 141 
moves along with the pointer 160. When the user turns 
on "Go Away", the sound object 141 moves away from 75 
the pointer 160. 

[0073] When the user has completed the above-men- 
tioned setting operations in step S3, the user goes on 
to step S4. In step S4, the user determines whether the 
above-mentioned setting operations have been per- 20 
formed on all desired sound files. If the decision is no, 
the user returns to step S1 and repeats the setting op- 
erations. 

[0074] In the above-mentioned examples, in the 
processing of step S1 , the user drags and drops the icon 25 
123 displayed in the file display block 122 of the sound 
file window 120 to select a sound file and performs the 
processing operations of steps S2 and S3 on the select- 
ed sound file. Besides this sound file selection method, 
the user may first select plural sound files in the stage 30 
115 and display the selected sound files as the sound 
objects 1 41 . Then, the user may select one of the sound 
objects 141 and perform the processing operations of 
steps S2 and S3 on the selected sound object 141. 
[0075] It should be noted that the processing opera- 35 
tions of steps S2 and S3 may be replaced each other. 
In addition, in the "Edit" setting, a screen may be pro- 
vided in which the sound object 141 is adapted to sound 
in response to a change other than that of brightness. 
Likewise, in the "Object" setting, a screen may be pro- *o 
vided in which another setting is made. 
[0076] Data such as the various parameters set as de- 
scribed above are stored as script data on the HDD 56 
or a recording medium not shown. Thereafter, the 
above-mentioned processing operations need not be 45 
repeated, thus enhancing the ease of use. The recorded 
data may be modified in parameter or replaced in sound 
file as required. The script data itself is compatible with 
a text file, so that the script data may be edited by a text 
editor for example. so 
[0077] If the user determines that the settings have 
been completed on all desired sound files, then the user 
goes on to step S5. In step S5, the user operates the 
command button 114 "PLAY". FIG. 15 shows an exam- 
ple in which three sound objects 141-1 through 141-3 55 
are displayed as a result of performing various settings 
on three selected sound files. 

[0078] As shown in the example of FIG. 15, when the 



user operates the "PLAY" command button, sound file 
names corresponding to the sound objects 141-1 
through 141-3 displayed on the stage 1 1 5 are displayed 
in the setting block 113. The sound objects 141-1 
through 141-3 are moving on the stage 115 according 
to the data set to them. These sound objects sound 
when the predetermined mesh of the image displayed 
in the image block 112 exceeds a preset brightness. The 
command button 114 "PLAY" is replaced by the "STOP" 
button, which the user presses to stop the above-men- 
tioned motion. 

[0079] The following describes other motions of the 
sound object 1 41 than described above, with reference 
to FIGS. 16A through 16C. As shown in these figures, 
the sound object 141 is shown as a circle. FIG. 16A 
shows a collision between the sound objects 141-1 and 
141-2. In this case, the sound objects 141-1 and 141-2 
bounce from each other (the travel directions of these 
sound objects change). The magnitude of this bounce 
is determined by the parameter set in the above-men- 
tioned "MASS" setting screen (FIG. 14A). 
[0080] FIG. 16B shows that the sound object 141 hits 
one of the screen frame (wall) of the stage 115 and 
bounces. Thus, the sound object 141 is set to bounce 
from the wall of the stage 115, so that no situation occurs 
in which the sound object 141 goes through the wall out 
of the stage 115 to disappear. However, if the stage 115 
is set as a three-dimensional space, the sound object 
141 is displayed smaller as it moves farther into the 
depth of the space. Consequently, the sound object 141 
may ultimately may look vanished from display. 
[0081] FIG. 16C shows that the user can drag the 
sound object 141 with the cursor 1 30. Thus, the present 
invention allows the user to directly control the motion 
of the sound object 141 . The user also make setting so 
that the sound object 141 dragged out of the stage 115 
will be deleted, thereby deleting all data associated with 
the sound object 141. 

[0082] Thus, only setting the basic motions of the 
sound object 141 allows the sound object 141 to perform 
various motions by selecting combinations of the basic 
motions. Consequently, the user can enjoy sounds not 
only audibly but also visually. 

[0083] The following describes a procedure of control- 
ling the displaying of the sound object 141 with refer- 
ence to FIG. 17. In step Sll, the user sets the sound ob- 
ject 141 to be controlled for display. In stepSl2, the user 
sets to the sound object 141 a parameter for controlling 
the displaying of the sound object 141 according to the 
above-mentioned display -control data already set by 
the user. 

[0084] If the user has just pressed command button 
114 "PLAY", the user sets the parameter for moving the 
sound object 141 in the direction set in the "Motion" 
screen (FIG. 11C). 

[0085] If the sound object 141 is already moving on 
the stage 115, then the user determines whether this 
sound object 141 has collided with another sound object 
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141 or an bubble generated by the pointer 160. If the 
decision is yes, then the user determines whether the 
bounce is to be displayed or not according to the data 
set in the "MASS" setting screen (FIG. 14A). If the 
bounce is to be displayed, the user set XYZ-coordinates s 
to which the bounced sound object 141 moves on the 
stage 115. 

[0086] This coordinates setting allows the user to set 
a parameter for changing the size of the sound object 
141 if the value of Z-coordinate changes. In the XYZ- io 
coordinates setting, the user also considers the magni- 
tude of the friction set in the "FRICTION" setting screen 
(FIG. 14A). Namely, if the magnitude of friction is large, 
the user must set the change in XYZ-coordinates to a 
relatively small level; if the magnitude of friction is small, *5 
the user must set the change in XYZ-coordinates to a 
relatively large level. 

[0087] If the motion for the pointer 160 has been set 
in the "Script" setting screen (FIG. 14C), the user sets 
a parameter such that the displaying is controlled ac- 20 
cording to the setting. 

[0088] Thus, when the user has set the parameters 
for controlling the displaying of the sound object 141, 
then, in step S1 3, the displaying of the sound object 141 
is controlled according to the parameters and a control 25 
result is shown on the stage 115. 
[0089] When the displaying of the sound object 141 
ends in step S13, then, back in step S11, the user per- 
forms the display control setting on another sound ob- 
ject 1 41 . The processing operations of step S1 2 and on 30 
are repeated. 

[0090] It should be noted that the processing de- 
scribed in this flowchart is ended when the command 
button 114 "STOP" for example is operated as an inter- 
rupt. 35 
[0091] The following describes how the sound object 
1 41 sounds in response to the brightness with reference 
to a flowchart shown in FIG. 18. In step S21 , an image 
sensed by the CCD video camera 23 is captured. The 
captured image data is sent to the processing block 82. 40 
In step S22, the processing block 82 executes feature 
extraction on the received image. The feature extraction 
performed here denotes tha extraction of brightness. 
[0092] The extracted brightness-associated data is 
sent through the graphics chip 81 to the microcontroller 45 
61. In step S23, the CPU 63 of the microcontroller 61 
checks, based on the brightness-associated data, for 
any mesh exceeding the brightness threshold set in the 
brightness setting screen (FIG. 11 A). If the decision is 
no, then, back in step S21, the processing operations 50 
up to step S23 are repeated. 

[0093] On the other hand, if the decision is yes in step 
S23, then the user sets in step S24 various parameters 
so that the sound object 141 generates a sound corre- 
sponding to a mesh found exceeding the brightness lev- 55 
el set in step S23. 

[0094] These parameters include the loudness of 
sound. The loudness of sound is associated with the 



size of the sound object 141 displayed on the stage 1 1 5. 
Namely, if the sound object 141 is displayed far in the 
depth of the stage 1 1 5 in a three-dimensional space and 
therefore the size of the sound object 141 is accordingly 
small, the loudness parameter is set so that the level of 
sound outputted from the sound object is accordingly 
low. 

[0095] Conversely, if the sound object 141 is dis- 
played forward on the stage 115 in a three-dimensional 
space and therefore the size of the sound object 1 41 is 
accordingly large, the loudness parameter is set so that 
the level of sound outputted from the sound object is 
accordingly high. If, for example, the sound object 141 
moves from back to forward on the stage 115, the loud- 
ness parameter is set so that the loudness gradually be- 
comes higher. 

[0096] If, for example, the sound object 141 moves 
from right to left on the stage 115, the parameter is set 
so that the sound moves from right to left, or a sound 
image is localized from right to left. Thus, the user sets 
the sound loudness and localization and the sound 
length. The sound length is set so that the sound object 
141 sounds for a time set in the sound length setting 
screen (FIG. 14B). 

[0097] When the user has set the above-mentioned 
sounding parameters, the sound object 141 generates 
the sound accordingly in step S25. Then, the processing 
operations of step S21 through step S25 are repeated. 
[0098] It should be noted that the processing de- 
scribed in this flowchart is ended when the command 
button 114 "STOP" for example is operated as an inter- 
rupt. 

[0099] The following describes an exemplary use 
method of an apparatus applied with the information 
processing apparatus according to the invention in 
which an image displayed on the LCD 21 changes ac- 
cording to an image taken by the CCD video camera 23 
and a sounded tone is changed accordingly. 
[0100] When the personal computer 1 is used as a 
wordprocessor for example, a tone to be sounded by 
the above-mentioned processing may be used as back- 
ground music and the sound object 141 displayed on 
the stage 115 as a screen saver. 

[0101] If the CCD video camera 23 is set such that the 
same shoots the user, the user can control the motion 
of the displayed sound object 141 and sound the same 
by user's motion. Consequently, the apparatus to which 
the inventive information processing apparatus is ap- 
plied can be used for live performance for example. This 
apparatus may also be used as a musical instrument. 
Further, if the CCD video camera 23 is set such that the 
same shoots a room door, a sound is generated in re- 
sponse to a person entering the room through the door. 
Consequently, this capability allows the user to set the 
apparatus used in a store for example such that a 
phrase "May I help you?" for example is sounded. 
[0102] Obviously, the information processing appara- 
tus according to the invention can be applied to other 
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than the personal computer 1 . The program providing 
medium for providing the computer program for execut- 
ing the above-mentioned processing includes network 
transmission media such as the Internet and a digital 
satellite in addition to the information recording media 
such as magnetic disc and CD-ROM. 
[01 03] While the preferred embodiment of the present 
invention has been described using specific terms, such 
description is for illustrative purposes only, and it is to 
be understood that changes and variations may be 
made without departing from the scope of the appended 
claims. 



Claims 



of said plurality of frames; and 
said reproducing means reproduces sound da- 
ta of any of said plurality of frames that exceeds 
said brightness threshold. 

5. An information processing method comprising the 
steps of: 

image-sensing an image of a subject; 
extracting predetermined feature data from 
said image sensed in the image-sensing step; 
setting sound data to be reproduced; and 
reproducing said sound data set in the setting 
step according to said feature data extracted in 
the extracting step. 



1. An information processing apparatus comprising: 

an image-sensing means for sensing an image 
of a subject; 

an extracting means for extracting predeter- 
mined feature data from said image sensed by 
said image-sensing means; 
a setting means for setting sound data to be re- 
produced; and 

a reproducing means for reproducing said 
sound data set by said setting means according 
to said feature data extracted by said extracting 
means. 

2. An information processing apparatus as claimed in 
claim 1, further comprising: 

a parameter setting means for setting a param- 
eter for controlling a motion of an object gener- 
ated in response to said sound data set by said 
setting means; and 

a display control means for controlling display- 
ing of said object according to said parameter 
set by said parameter setting means and said 
feature data extracted by said extracting 
means. 

3. An information processing apparatus as claimed in 
claim 2, further comprising: 



6. An information processing method as claimed in 
claim 5, further comprising the steps of: 

20 setting a parameter for controlling a motion of 

an object generated in response to said sound 
data set in the setting step; and 
controlling displaying of said object according 
to said parameter set in said parameter setting 

25 step and said feature data extracted in the ex- 

tracting step. 

7. An information processing method as claimed in 
claim 6, further comprising the step of: 

30 

recording said data set in the setting step and 
said parameter set in the parameter setting 
step. 

35 8. An information processing method as claimed in 
claim 5, wherein said feature data extracted in the 
extracting step is data associated with brightness; 

the setting step divides said image sensed in 
the image-sensing step into a plurality of 
frames and sets a brightness threshold to each 
of said plurality of frames; and 
the reproducing step reproduces sound data of 
any of said plurality of frames that exceeds said 
45 brightness threshold. 



a recording means for recording said data set 
by said setting means and said parameter set 
by said parameter setting means. 

so 

4. An information processing apparatus as claimed in 
claim 1 , wherein said feature data extracted by said 
extracting means is data associated with bright- 
ness; 

55 

said setting means divides said image sensed 
by said image-sensing means into a plurality of 
frames and sets a brightness threshold to each 



9. An information providing medium for providing a 
program readable by a computer for making an in- 
formation processing apparatus execute process- 
ing including the steps of: 

image-sensing an image of a subject; 
extracting predetermined feature data from 
said image sensed in the image-sensing step; 
setting sound data to be reproduced; and 
reproducing said sound data set in the setting 
step according to said feature data extracted in 
the extracting step. 
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10. An information providing medium as claimed in 
claim 9, said processing further including the steps 
of: 

setting a parameter for controlling a motion of 5 
an object generated in response to said sound 
data set in the setting step; and 
controlling displaying of said object according 
to said parameter set in said parameter setting 
step and said feature data extracted in the ex- io 
tracting step. 

11. An information providing medium as claimed in 
claim 10, said processing further including the step 

of: 15 

recording said data set in the setting step and 
said parameter set in the parameter setting 
step. 

20 

12. An information providing medium as claimed in 
claim 9, wherein said feature data extracted in the 
extracting step is data associated with brightness; 

the setting step divides said image sensed in 25 
the image-sensing step into a plurality of 
frames and sets a brightness threshold to each 
of said plurality of frames; and 
the reproducing step reproduces sound data of 
any of said plurality of frames that exceeds said 30 
brightness threshold. 

1 3. A program readable by a computer for making an 
information processing apparatus execute process- 
ing including the steps of: 35 

image-sensing an image of a subject; 
extracting predetermined feature data from 
said image sensed in the image-sensing step; 
setting sound data to be reproduced; and *o 
reproducing said sound data set in the setting 
step according to said feature data extracted in 
the extracting step. 

45 
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